Digital signage for immersive views

ABSTRACT

Techniques for generating and superimposing digital signage onto an immersive view. In an exemplary embodiment, certain entities are occluded or otherwise unviewable in an immersive view of an environment. For such entities, digital signage may be generated and superimposed in a suitable area of the immersive view, and further indicating, e.g., directions to such entities from a viewing location, as well as other metadata, such as business name, website, etc. To determine whether a nearby entity is displayed in the immersive view, a planar layout storing geometrical data associated with entities in the vicinity may be loaded and processed.

BACKGROUND

Systems generating immersive views of designated environments are seeingwidespread adoption in state-of-the-art Internet and mobileapplications. Such systems may generate images that effectively“immerse” the user in any number of environments, e.g., street-sidescenery, indoor areas, manufacturing or medical scenarios, etc., fromthe vantage point of a viewing location as specified by the user.Current technologies may further superimpose limited data, e.g.,business names, store hours, and other digital “signage,” on objectsvisible in the immersive view. However, entities of interest not withinthe field of view may not be recognized or indicated.

For example, a user may type an address corresponding to a bank, and theimage system may deliver an immersive view corresponding to a viewerstanding in front of the bank. However, current systems may not supportplacing digital signage indicating out-of-view entities, e.g., anotherbank directly behind the viewer, or other locations of interest aroundthe corner that are not directly within the line-of-sight.

Accordingly, it would be desirable to provide techniques for an imagegeneration system to generate and place digital signage corresponding toout-of-view entities of interest in an immersive view.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Briefly, various aspects of the subject matter described herein aredirected towards techniques for indicating entities of interest in animmersive view, and further providing information associated with suchentities of interest. In certain aspects, the view of an entity ofinterest may be obstructed by one or more occlusions, and suchout-of-view entities may be identified using geometrical, geographical,or other planar layout data associated with an environment of theviewer. Digital signage containing data corresponding to the entities ofinterest may then be generated and superimposed onto a suitable canvasarea of the immersive view.

Other advantages may become apparent from the following detaileddescription and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1-2 illustrate instances of an immersive view of certainstreet-side scenery, e.g., corresponding to a user-input street address,and a standard planar map corresponding to the environment shown in theimmersive view.

FIG. 3 illustrates digital signage superimposed on an immersive view,positioned adjacent to a building.

FIG. 4 shows an example of an immersive view generated according totechniques of the present disclosure.

FIG. 5 shows an exemplary embodiment of an image generation systemaccording to the present disclosure.

FIG. 6 illustrates an exemplary embodiment of block 540 in FIG. 5.

FIG. 7 illustrates an exemplary embodiment of block 610 in FIG. 6.

FIG. 8 illustrates an example of a planar layout corresponding to theillustrative immersive view described hereinabove with reference toFIGS. 1 and 2.

FIG. 9 illustrates an alternative example of a multi-level planarlayout, corresponding to an environment inside a multi-level building,such as a multi-level shopping mall.

FIG. 10 illustrates an example of a dynamic occlusion.

FIG. 11 illustrates an exemplary embodiment of an apparatus according tothe present disclosure.

FIG. 12 illustrates an exemplary embodiment of a computing deviceaccording to the present disclosure.

FIG. 13 illustrates an exemplary embodiment of a method according to thepresent disclosure.

DETAILED DESCRIPTION

Various aspects of the technology described herein are generallydirected towards techniques for generating immersive view of auser-specified environment or locale. It will be appreciated thatcertain features of the techniques described below may be used for anysystems for generating immersive views, e.g., street-side views, indoorviews, etc., and may find application in any types of imaging systems,e.g., virtual or augmented reality, telepresence, digital mapapplications, electronic or video gaming applications, etc. Suchalternative exemplary embodiments are contemplated to be within thescope of the present disclosure.

The detailed description set forth below in connection with the appendeddrawings is intended as a description of exemplary means “serving as anexample, instance, or illustration,” and should not necessarily beconstrued as preferred or advantageous over other exemplary aspects. Thedetailed description includes specific details for the purpose ofproviding a thorough understanding of the exemplary aspects of theinvention. It will be apparent to those skilled in the art that theexemplary aspects of the invention may be practiced without thesespecific details. In some instances, well-known structures and devicesare shown in block diagram form in order to avoid obscuring the noveltyof the exemplary aspects presented herein.

Modern image generation systems have the capability to digitallygenerate an immersive view of an environment according to user-specifiedparameters. The immersive view provides a user with the experience ofbeing “immersed” in the specified environment, from a user-definedvantage point. Such systems may further allow for user interactionwithin the immersive view, e.g., to change the angle at which scenery isperceived by panning or tilting the viewing vantage point, and/oradvancing the scenery along predetermined directions, etc.

FIG. 1 illustrates an immersive view 100 of certain street-side scenery,corresponding to, e.g., a user-input street address. FIG. 2 illustratesa standard planar map 200 corresponding to the environment shown inimmersive view 100. It will be understood that FIGS. 1-2 are shown forillustrative purposes only, and are not meant to limit the scope of thepresent disclosure to any particular types of views or scenery that canbe processed using the technologies disclosed herein. Aspects of thepresent disclosure may accommodate, e.g., immersive views of street-sidescenery, indoor areas, manufacturing or medical environments,telepresence applications, etc.

In FIG. 1, immersive view 100 shows various geographical items of note,such as buildings 110, 120, crosswalk 150, etc., from a designatedviewing vantage point. Map 200 in FIG. 2 shows the same objects markedon a standard planar map as elements 210, 220, 250, respectively.Navigation icon 160 in FIG. 1 shows user-interactive controlssuperimposed onto immersive view 100 to allow the user to, e.g., changethe viewing location, or angle from which immersive view 100 isgenerated. For example, by dragging the curved arrow in icon 160 in agiven direction, the angle at which immersive view 100 is generated maybe panned up and down, sideways, etc.

It will be appreciated that the correspondence between immersive view100 and planar map 200 is generally determined by certain parameters,also denoted herein as “viewing parameters,” associated with a viewingvantage point 290, illustratively marked with an “X” on map 200 in FIG.2. In particular, viewing parameters may include, e.g., geographicalcoordinates (such as latitude/longitude, GPS coordinates, etc.), viewingorientation (illustratively indicated by vector 292), viewing elevation,one or more angles of view (illustratively denoted in FIG. 2 as “a”),etc. Thus immersive view 100 may correspond to the view of anenvironment that would be perceived by a human observer from thedesignated viewing vantage point 290, and further with the other viewingparameters as specified with vantage point 290.

In some implementations, the parameters of viewing vantage point 290 maybe directly specified by a user of the system. For example, the user maytype an address, latitude/longitude, or any other coordinatesidentifying the location of a specific vantage point, or “viewinglocation.” Alternatively, certain of the viewing parameters may beinferred by the image generation system, e.g., based on sub-modulescoupled to the image generation capable of detecting viewingorientation, current user location (e.g., in an application wherein a“user” of the image generation system is also identified with the“viewer” of an immersive view) using GPS, etc.

To generate an immersive view, an image generation system may access astored digital database of real-life imagery, e.g., as obtained fromdigital photographs of a plurality of pre-specified environments, and/orpost-processed versions of such photographs. Furthermore, an imagegeneration system may display additional information associated with anyobjects identifiable by the system in a particular immersive view. Forexample, metadata relating to identified objects in view 100 may bedigitally superimposed onto view 100, e.g., with metadata being placedadjacent to corresponding objects. Metadata may include, e.g., the namesof businesses resident at a building, store hours, associated websites,paid-for advertising, etc.

For example, FIG. 3 illustrates digital signage 310 superimposed onimmersive view 300, positioned adjacent to building 120. Digital signage310 shows, e.g., the name (“First Bank,” as also illustrativelyindicated in FIG. 2 at element 220) of a business resident in building120, along with information on business hours and associated website.

While digital signage 310 may readily be provided adjacent to building120, which is located within a direct line of sight (LOS) of the viewingvantage point, it is more challenging to identify and place digitalsignage corresponding to entities not within the field of view. Forexample, note building 230 in FIG. 2 (illustratively labeled “Bob's DryCleaner”) is not displayed in immersive view 100 in FIG. 1, because itsview is blocked by building 110. In this case, the view of building 230is designated as being “occluded” by building 110. Accordingly, thepresence of building 110 as a (static) occlusion in the direct line ofsight between viewing vantage point 290 and building 230 preventsdigital signage for “Bob's Dry Cleaner” from being readily placedadjacent to building 230 in immersive view 100.

Alternatively, a building such as building 235 may also not be displayedin immersive view 100, as such a building lies outside the angle of viewa defined by the viewing parameters. In both cases (i.e., when the viewof an entity is occluded, or when an entity lies outside an angle ofview defined by the viewing parameters), an entity is not displayed inimmersive view 100, even though such entity may lie in the vicinity ofthe viewer.

Accordingly, it would be desirable to provide techniques for identifyingentities of interest that are not displayed in an immersive view, and togenerate and display digital signage for such entities in the immersiveview.

FIG. 4 shows an illustrative immersive view 400 generated according totechniques of the present disclosure. In FIG. 4, digital signage 410includes information for building 230 (e.g., “Bob's Dry Cleaner”), whichwas not originally within the viewer line of sight in immersive view100. In the exemplary embodiment shown, digital signage 410 furtherillustratively includes directions to building 230. In an exemplaryembodiment, the directions provided in digital signage 410 can bereferenced from a location corresponding to signage 410 in immersiveview 400 (e.g., “walk 50 meters due left”). In an alternative exemplaryembodiment, directions can alternatively be referenced from the viewingvantage point, e.g., vantage point 290 as labeled in FIG. 2 (e.g., “walk10 meters straight ahead, then turn left and walk 50 meters”). In anexemplary embodiment, digital signage 410 may include an arrow 411pointing in the direction of the out-of-view entity.

By identifying and displaying digital signage for entities not displayedin an immersive view, the techniques of the present disclosureadvantageously enhance the range of offerings and/or other objects ofinterest (also denoted herein as “preferred entities”) that may beindicated to a viewer in the vicinity of the viewing location. Furthernote in alternative exemplary embodiments, out-of-view digital signagemay also be superimposed for preferred entities not located in theimmediate vicinity of the viewing location. For example, if a userinputs to the image generation system the name of a specific serviceprovider (e.g., a bookstore), then the system may display digitalsignage corresponding to preferred entities offering similar types ofservices that need not reside within the immediate vicinity of theviewer (e.g., the web address of an online bookstore). Such alternativeexemplary embodiments are contemplated to be within the scope of thepresent disclosure.

FIG. 5 shows an exemplary embodiment 500 of an image generation systemaccording to the present disclosure. Note FIG. 5 is shown forillustrative purposes only, and is not meant to limit the scope of thepresent disclosure to any particular implementations or combinations offeatures described herein for generating an immersive view. For example,in alternative exemplary embodiments, a system may include only a subsetof the functional blocks shown in FIG. 5.

Furthermore, FIG. 5 is not meant to suggest that the digital signage forout-of-view entities such as shown in FIG. 4 is necessarilyautomatically generated by a computing system according to techniques ofthe present disclosure. In alternative exemplary embodiments, digitalsignage such as 410 in FIG. 4 may be manually created and entered into adatabase, e.g., by a sponsor of a preferred entity, and further manuallyplaced in certain immersive views prior to the time of user interactionwith the image generation system. A computer implementing the imagegeneration system may then digitally superimpose the manually positioneddigital signage onto digital immersive views at the same time as orprior to when a user requests the generation of an immersive view. Suchalternative exemplary embodiments are contemplated to be within thescope of the present disclosure.

In FIG. 5, at block 510, viewing parameters 510 a of a viewerperspective are determined In an exemplary embodiment, viewingparameters 510 a are sufficient to specify a unique immersive view,e.g., immersive view 100, of an environment for display. Viewingparameters 510 a may identify the viewing location within a geographicalor topological frame of reference, and may include, e.g., geographicalcoordinates of user location such as latitude and longitude, GPScoordinates, street address, name or identifier of a business having aunique geographical location, etc. Viewing parameters 510 a may furtherinclude an orientation in which the viewer's gaze is directed, viewingangle, altitude angle, etc. Viewing parameters 510 a may also include avertical elevation of the viewer, e.g., viewing elevation, if a vieweris standing at a raised or lowered elevation relative to surroundinglandscape, etc.

In an exemplary embodiment, parameters 510 a may be partially input tothe system by the user, e.g., by typing in an address in a search bar,etc. In an alternative exemplary embodiment, the system mayautomatically derive at least some of viewing parameters 510 a from,e.g., a GPS subsystem of the hardware on which the system is running,cellular base stations, etc. In an exemplary embodiment wherein thepresent techniques are incorporated in, e.g., a mobile device, agyroscope or other directional determining subsystem on the mobiledevice may be utilized to provide information on viewing parameters,e.g., viewing orientation. Such alternative exemplary embodiments arecontemplated to be within the scope of the present disclosure.

At block 520, one or more preferred entities 520 a may be identifiedbased on viewing parameters 510 a. In particular, viewing parameters 510a may specify a viewing location, and the one or more preferred entities520 a may correspond to an entity in the vicinity of the viewinglocation. Alternatively, the one or more preferred entities 520 a maycorrespond to an entity not necessarily in the vicinity of the viewinglocation, but related to some functional attribute with an entityspecified in viewing parameters 510 a, e.g., being a restaurant offeringa similar type of food to a restaurant specified in viewing parameters510 a, etc.

In an exemplary embodiment, a preferred entity may correspond to anyentity for which it is desired to place corresponding digital signage inimmersive view 530 a, e.g., containing information relevant to suchpreferred entity. In an exemplary embodiment, the identity of such apreferred entity may be, e.g., designated by the user, assigned by thesystem, or derived from both user input and system processing.

For example, in an illustrative scenario wherein the preferred entity isdesignated by the user, the user could input the address for a desiredlocation, e.g., a business, a residence, park, landmarks, etc., and suchdesired location may correspond to a preferred entity. Alternatively,the image generation system may store predetermined locations that liewithin a general vicinity of a viewing location, e.g., as determined atblock 510, that could be relevant to the user. For example,system-assigned preferred entities may include landmarks of interestsuch as banks, parks, entertainment venues, and stores. Such entitiesmay be based on user-specified preferences configured, e.g., at systeminitialization. In an exemplary embodiment, the system identifiesbusinesses within the viewing vicinity that are similar or otherwiserelate to a business originally designated by the user.

System-assigned preferred entities may further include preferred localor other businesses that may be relevant to the user at the specifiedviewing location. In an exemplary embodiment, the assignment of abusiness as a preferred entity may be driven by an advertising model,e.g., businesses may pay advertising fees to be identified as apreferred entity to certain users (e.g., users having specified matchingpreferences, or within a geographical vicinity of some area, etc.) orall users of the system. Such alternative exemplary embodiments arecontemplated to be within the scope of the present disclosure.

Further based on the parameters 510 a of viewer perspective asdetermined at block 510, at block 525, a planar layout 525 a associatedwith the vicinity of the viewing location is loaded, e.g., into memoryfor processing. Planar layout 525 a may correspond to, e.g., horizontaland/or vertical layout, location, and/or other geometrical dataassociated with the geographical environs of the viewing location.Planar layout 525 a will generally specify corresponding heightinformation for the entities present in the layout, and may thus beutilized to determine the presence of static occlusions in the linesight of a viewer to a chosen entity. Planar layout 525 a may furtherspecify identifying metadata associated with entities present in thelayout, e.g., business names, business types, street addresses, etc.

Note planar layout 525 a may generally encompass any type ofenvironment, e.g., indoors or outdoors, urban or suburban or rural,single-level (e.g., a room in a house) or multi-level (e.g., one floorof a multi-floor indoor or outdoor shopping mall, subterranean), etc.Such alternative exemplary embodiments are contemplated to be within thescope of the present disclosure.

FIG. 8 illustrates an example 800 of a planar layout corresponding tothe illustrative immersive view described hereinabove with reference toFIGS. 1 and 2. Note FIG. 8 is shown for illustrative purposes only, andis not meant to limit the scope of the present disclosure to anyparticular types of scenery or environments that can be accommodated.For illustrative purposes, entities labeled “8xx” in FIG. 8 maycorrespond to objects labeled “1xx” in FIG. 1, unless otherwise noted.

In FIG. 8, layout 800 includes multiple entities 810, 820, 830, 860.Each entity may correspond to, e.g., a static object of interest inlayout 800, and each object may or may not occupy some planar spaceand/or vertical height. Examples of entities may include, e.g., localbusinesses, parks, roads, user-designated locations of interest, fixedobjects such as trees, etc. Each entity may further be associated withgeometrical data that specifies the physical dimensions of that entity,e.g., height, width, perimeter, radius, etc. Note in alternativeexemplary embodiments (not shown), some entities need not occupy anyvertical height, e.g., an entity may be separately associated withcrosswalk 250 in FIG. 2. Furthermore, entities may also includeinformation associated with objects not occupying any space, e.g., adesignation of a point in planar layout 800 as corresponding to “TheCity of San Francisco” for reference purposes.

For example, in FIG. 8, entity 810, corresponding to building 110 inFIG. 1, is indicated as having associated surface area of 100 m×80 m,and a vertical height of 150 m. Similarly, entities 820 and 830 may haveassociated geometrical dimensions as indicated in FIG. 8. Note thesurface area of entity 860 is expressed as having a radius of 2 m, and avertical height of 10 m. Entity 860 may correspond to, e.g., a tree thatis present in layout 800, and the presence of such entities in generalmay be noted in the layout 800 to aid in determination of the presenceof static occlusions, as further described hereinbelow with reference toFIGS. 6 and 7.

Note the specification formats of geometrical data in FIG. 8 are shownfor illustrative purposes only, and is not meant to limit the scope ofthe present disclosure. For example, in alternative exemplaryembodiments, a planar layout may adopt alternative ways to specify thegeometrical dimensions of an entity such as tree 860 or building 810,e.g., with alternative dimensional specifications (e.g., round versusstraight lines), alternative constituent shapes, finer or coarserspatial granularity, and/or alternative numerical representations thanare shown in FIG. 8. Such alternative exemplary embodiments arecontemplated to be within the scope of the present disclosure.

Note entity 810 may correspond to building 110 in FIG. 1. It will beappreciated, however, that entity 810 need not correspond to all ofbuilding 110, and may merely correspond to e.g., a sub-section or somefloors thereof. Other entities (not shown) may also be mapped to thespatial area occupied by building 110, e.g., other floors having otheroccupants, etc. To distinguish among multiple entities in a multi-storybuilding, geometrical data may generally specify both horizontal andvertical (e.g., vertical elevation) location and spatial extent (e.g.,the floor-to-ceiling height of an office) information associated witheach entity.

FIG. 9 illustrates an alternative example of a multi-level planar layout900, corresponding to an environment inside a multi-level building, suchas a multi-level shopping mall. In FIG. 9, the viewing location isillustratively on the 2nd floor of the building, as marked by locationmarker 990 in 2nd floor planar layout 902. Planar layout 900 furtherincludes the planar layout 904 for a 3rd floor. Entities 910-960 areshown, each corresponding to, e.g., a stall or shop space, etc., inplanar layouts 902, 904. Each of entities 910-960 may have associatedgeometrical data (not shown in FIG. 9), as similarly described withreference to entities 810-830 in FIG. 8.

In an exemplary embodiment, planar layouts 902, 904 may further containassociated information (not shown in FIG. 9) that represents therelative vertical separation and horizontal alignment between entitieson the 2^(nd) and 3^(rd) floors. It will be appreciated that planarlayouts encapsulating all such single- and multi-level floor and/orstreet and/or other environments are contemplated to be within the scopeof the present disclosure.

Referring back to FIG. 5, at block 530, a digital image 530 a for animmersive view corresponding to viewing parameters 510 a is loaded intomemory. Digital image 530 a may also be denoted herein as an“unannotated immersive view,” as distinguished from, e.g., (annotated)immersive view 550 a generated by block 550, as further describedhereinbelow.

In an exemplary embodiment, digital image 530 a may correspond to, e.g.,one of multiple images stored or generated for each of a plurality ofreference points and/or viewing angles (not shown) on planar layout 525a. In an exemplary embodiment, such multiple images corresponding to areference point may collectively constitute a 360-degree panoramic viewof scenery as viewed from the perspective of that reference point. Suchmultiple images may be accessed from a pre-configured database, eitheronline or offline. Based on, e.g., viewing parameters 510 specifyingviewing location and viewing orientation as determined, the correctimage 530 a from the multiple stored images may be retrieved.

Referring back to FIG. 5, block 540 generates digital signage 540 a tobe superimposed onto unannotated immersive view 530 a. In an exemplaryembodiment, the positions in planar layout 525 a corresponding toviewing location (e.g., as specified in parameters 510 a) and preferredentity 520 a are individually determined. Furthermore, the field of viewcovered by immersive view 530 a in planar layout 525 a may be determined

For example, referring to FIGS. 1 and 8, illustrative immersive view 100may be determined at block 540 to correspond to a field of view 808(e.g., the area within the dark bold line) of planar layout 800 in FIG.8.

At block 550, based on the information in 520 a, 525 a, 530 a, digitalsignage 540 a is generated. In an exemplary embodiment, digital signage540 a may incorporate and/or display, e.g., data and/or metadatacorresponding to preferred entity 520 a. Digital signage 540 a mayfurther include data or information specific to the location ofpreferred entity 520 a with reference to immersive view 530 a. Forexample, digital signage 540 a may indicate the directions a viewerwould follow to go from the current viewing location to arrive atpreferred entity 520 a.

Note such directions may be generated, e.g., using knowledge of planarlayout 525 a, as further described hereinbelow with reference to FIGS. 6and 7. Alternatively, directions or other information to be included indigital signage 540 a may be generated using other techniques, e.g.,derived from manual commenting and annotation of specific immersiveviews 530 a. In an exemplary embodiment, the generated directions mayaccount for cases in which preferred entity 520 a is occluded inimmersive view 530 a. Such directions may specify, e.g., that one wouldhave to go around or otherwise bypass one or more occlusions to reachthe preferred entity 520 a. Techniques for determining directions from aviewing location to an occluded entity are further described hereinbelowwith reference to FIGS. 6 and 7. Note, however, that such techniques aredescribed for illustrative purposes only, and are not meant to limit thescope of the present disclosure to exemplary embodiments necessarilyemploying such techniques.

At block 550, digital signage 540 a is superimposed on immersive view530 a to generate annotated immersive view 550 a. In an exemplaryembodiment, to generate immersive view 550 a, block 550 may determine anoptimal area of digital image 530 a in which to place the generateddigital signage 540 a. For example, in cases wherein preferred entity520 a is within line of sight, then corresponding digital signage 540 amay be placed directly adjacent to the image of preferred entity 520 ain immersive view 550 a. Alternatively, in cases wherein preferredentity 520 a is occluded, or otherwise outside the field of view, thencorresponding digital signage 540 a may be placed in an area ofimmersive view 550 a that is indicative of a direction to take to reachpreferred entity 520 a from the viewing location.

Further considerations for placement of digital signage 540 a mayinclude the availability of suitably sized “blank” or “non-critical”space in immersive view 550 a, e.g., corresponding to open sky, a wallof a building, an empty billboard, a billboard on the side of a truckfor truckside advertising, etc. Such space may be automaticallydetermined, e.g., using image recognition and pattern matchingtechniques applied to immersive views 530 a, and/or utilizing objectdimensional information as further described hereinbelow with referenceto FIG. 10. Alternatively, space for digital signage 540 a placement maybe manually determined, e.g., by crowd-sourcing, or based on input fromadvertisers, vendors, image generation system designers, etc.

Note while blocks 540 and 550 are separately shown in FIG. 5 forillustrative purposes, it will be appreciated that blocks 540 and 550may generally be integrated into one functional block in alternativeexemplary embodiments. For example, digital signage 540 a may includedirections to preferred entity 520 a that depend on where digitalsignage 540 a is ultimately placed in immersive view 550 a, and thisfunctional inter-relation may thus be accounted for in computationsperformed by the image generation system jointly for blocks 540 and 550.Such alternative exemplary embodiments are contemplated to be within thescope of the present disclosure.

FIG. 6 illustrates an exemplary embodiment 540.1 of block 540 in FIG. 5.Note FIG. 6 is shown for illustrative purposes only, and is not meant tolimit the scope of the present disclosure to any specific implementationof block 540 shown herein.

In FIG. 6, at block 610, it is determined whether the preferred entityis in the line of sight (LOS) from the viewer perspective, e.g., asspecified by parameters 510 a. In an exemplary embodiment, geometricdata as obtained from planar layout 525 a may be used to determine ifpreferred entity 520 a is within the viewer's line of sight. Note anexemplary embodiment 610.1 of block 610 is further described hereinbelowwith reference to FIG. 7 for illustrative purposes.

If the preferred entity is in the LOS, the system proceeds to block 625.If not, the system proceeds to block 620.

At block 625, if the preferred entity is within the LOS, then a “canvasarea” is determined in the immersive view 530 a. The canvas area maycorrespond to the position in immersive view 530 a where digital signage540 a associated with preferred entity 520 a will be superimposed. In anexemplary embodiment, when the preferred entity is in the LOS, thecanvas area may be chosen to be, e.g., directly adjacent to thepreferred entity in immersive view 525 a.

Following block 625, at block 655, an exemplary embodiment 540.1 a ofdigital signage 540 a, is generated. Note digital signage 540 a mayinclude, e.g., metadata associated with preferred entity 520 a.

At block 620, if preferred entity 520 a is not within the LOS, then a“canvas area” is determined in immersive view 530 a. Note when preferredentity 520 a is not in the LOS, the canvas area may be chosen to be,e.g., in an “empty” area of immersive view 530 a (e.g., any areanon-overlapping with existing entities in view 530 a), or any otherblank or non-critical space as earlier described hereinabove withreference to FIG. 5.

In an exemplary embodiment, certain techniques may be utilized fordetermining an optimal canvas area in an immersive view for placement ofdigital signage, e.g., such as described with reference to blocks 620,625. For example, immersive view 530 a may be further associated withthree-dimensional (3D) data describing properties of objects present inimmersive view 530 a, e.g., shapes, separation distances, etc. Such 3Ddata may be generated from a variety of sources, e.g., lidar,vision-based geometry, street footprints, etc. 3D information may thusbe specified to identify the presence of “dynamic” occlusions in animmersive view, e.g., occlusions that appear in immersive view that arenot expected to be present when a viewer is actually physically presentat the viewer location and viewing the specified scene.

For example, referring to FIG. 3 hereinabove, block 625 may identify thearea shown for placement of digital signage 310, in the absence of anystatic occlusions in the line of sight between the viewer position andbuilding 120. However, referring to FIG. 10, a dynamic occlusion such asa large truck (illustratively represented in FIG. 10 as a blankrectangle 1010) making a left turn onto street 204 from street 202 mayexist in immersive view 1000. Dynamic occlusion 1010 may not be presentin planar layout 800 due to its dynamic nature, and occlusion 1010 mayfurther happen to overlap with the area corresponding to building 120 inthe immersive view, as well as the area where digital signage 310 isshown as being placed in FIG. 3.

In such an instance, block 625 may take any of several steps to addressthe situation. In a first scenario, if it is determined that a dynamicocclusion occludes a significant portion of an immersive view, thendigital signage may simply not be displayed in the immersive view. Thedetermination that the dynamic occlusion occludes a significant portionof an immersive view may be made, e.g., using data obtained duringphotography capture of the real-life scenes (such as data on separationdistances from lidar, vision-based geometry, street footprints, etc.,wherein a small separation distance between a camera and the occludingobject would indicate that the occlusion may occupy a significantportion of the view), or from manual annotation by post-review ofimmersive views prior to upload to the system.

In a second scenario, if is determined that a dynamic occlusion (such asocclusion 1010 in FIG. 10) overlaps with an area that would be otherwisechosen for placement of digital signage, then the digital signage may beplaced in an alternative canvas area. For example, digital signage forbuilding 120 may be placed, e.g., slightly higher or more to the rightthan is currently shown for digital signage 310 in FIG. 3, to avoidoverlap with the dynamic occlusion. It will be appreciated thattechniques for determining such area overlap may account for, e.g., 3Dinformation characterizing the spatial extent, position, and/or distanceto camera of dynamic occlusion 1010 in immersive view 1000 (e.g., asobtained during photography capture of the immersive view). Suchalternative exemplary embodiments are contemplated to be within thescope of the present disclosure.

In a third scenario, digital signage may still be placed adjacent to acorresponding entity even if a dynamic occlusion exists. In particular,digital signage may be placed directly on top of the dynamic occlusion.

In an exemplary embodiment, the presence of dynamic occlusions may bedetermined using 3D data (e.g., on separation distances, geometry, etc.,generated from a variety of sources, e.g., lidar, vision-based geometry,street footprints, etc.) during photography capture of the immersiveviews, or determined during post-processing of the immersive views bymanual or computer vision/pattern matching techniques. Theidentification of a dynamic occlusion may further be made by comparisonand/or correlation with a planar layout 525 a, e.g., to identify objectspresent in an immersive view but not present in a corresponding planarlayout. Such alternative exemplary embodiments are contemplated to bewithin the scope of the present disclosure.

In an exemplary embodiment, the canvas area may alternatively or furtherbe manually specified, e.g., by a sponsor of the preferred entity. Forexample, a representative of the proprietor of “First Bank,”corresponding to building 220 in FIG. 2, or of “Bob's Dry Cleaner,”corresponding to building 230 in FIG. 2, may directly specify to theimage system that its associated digital signage should be placed in animmersive view at the positions shown in FIG. 3 or 4. In an exemplaryembodiment, certain parameters of digital signage may thus be manuallyspecified, e.g., the placement of digital signage within the immersiveview, placement relative to other digital signage, size of the digitalsignage, etc. Such alternative exemplary embodiments are contemplated tobe within the scope of the present disclosure.

At block 630, immersive view 530 a is correlated to a correspondingfield of view in planar layout 525 a. This position may be determinedby, e.g., accounting for viewing parameters 510 a (e.g., specifyingviewing location and viewing angle in planar layout 520 a). For example,as illustrated in FIG. 8, field of view 808 may correspond toillustrative immersive view 100, as earlier described hereinabove withreference to FIG. 5.

At block 640, directions from the canvas area or viewing location inimmersive view 530 a to non-LOS preferred entity 520 a are determined.In an exemplary embodiment, directions may include distance to betraveled, turns (e.g., right or left) to be taken, time required, etc.,by a viewer initially starting from the specified viewing location.

At block 650, an exemplary embodiment 540.1 a of digital signage 540 ais generated for the non-LOS preferred entity. In an exemplaryembodiment, the digital signage may include metadata associated with thepreferred entity, and/or directions to preferred entity 520 a, e.g., asdetermined at block 640.

FIG. 7 illustrates an exemplary embodiment 610.1 of block 610 in FIG. 6.Note FIG. 7 is shown for illustrative purposes only, and is not meant tolimit the scope of the present disclosure to any particular techniquesfor detecting whether an entity is within LOS of a viewer.

In FIG. 7, at block 705, it is determined whether preferred entity 520 ais within a field of view of immersive view 530 a. In an exemplaryembodiment, this determination may be performed by comparing, in theplanar layout 525 a, the field of view defined by immersive view 530 awith the location of preferred entity 520 a. If preferred entity 520 ais within the field of view, then the system may proceed to block 710.Otherwise, preferred entity 520 a may be determined to be not within LOSat block 715.

For example, referring to FIG. 9, an indoor immersive view (not shown)of the second floor of a shopping mall having planar layout 900 maycorrespond to a field of view 908. Note in FIG. 9, field of view 908 maycorrespond to an area of the second floor 902 bordered by the dark boldline, wherein dotted segments 908a, 908b indicate that field of view 908lies entirely on the second floor, and are effectively seen through a“transparent” layout of third floor 904. In this case if preferredentity 520 a corresponds to, e.g., stall 950 on third floor 904, then itmay be determined that entity 950 lies on the third floor, while fieldof view 908 is restricted to the second floor. Accordingly, preferredentity 520 a is determined to lie outside field of view 908 on thesecond floor planar layout 902 defined by the immersive view.

In an exemplary embodiment, for a multi-level structure such asdescribed with reference to FIG. 9, fields of view may be automaticallydefined to include no more than the area associated with a single floor,e.g., assuming that the floor/ceiling separating two stories in amulti-level building is opaque. In an alternative exemplary embodiment,the determination of LOS status in a multi-level structure mayalternatively or in conjunction utilize “direct line calculation”techniques, e.g., as described hereinbelow with reference to blocks720-725. Such alternative exemplary embodiments are contemplated to bewithin the scope of the present disclosure.

Returning to FIG. 7, at block 710, a direct line from the viewinglocation (e.g., as specified in parameters 510 a) to the preferredentity 530 a is calculated in planar layout 520 a.

For example, for the illustrative planar layout 800 shown in FIG. 8described hereinabove, direct line 801 may be drawn between viewinglocation 290 and illustrative preferred entity 830. Alternatively, forthe illustrative planar layout 900 shown in FIG. 9 describedhereinabove, direct line 901 may be drawn between viewing location 990and illustrative preferred entity 950.

At block 720, it is determined whether the direct line intersects withany blocking entities, based on planar layout 520 a. In an exemplaryembodiment, geometric data, e.g., vertical height data, associated withentities intersecting direct line 710 a may be utilized to determinewhether such entities would constitute a “blocking entity.”

For example, for the illustrative planar layout 800 shown in FIG. 8,direct line 801 intersects both entity 810 and entity 860. The fact thatthe vertical height of entity 810 is 150 meters, and the vertical heightof entity 860 is 10 meters, may be used to establish the presence of twoblocking entities.

Alternatively, for the illustrative planar layout 900 shown in FIG. 9,direct line 901 between the second floor 902 and third floor 904intersects a “third level floor” entity 906 at point 905. Accordingly,the presence of the blocking entity is established.

If, at block 720, it is determined that there exist one or more blockingentities, then the system proceeds to block 725, where it is determinedthat no LOS exists. Alternatively, if it is determined that there existsno blocking entities, then the system proceeds to block 730, where it isdetermined that an LOS exists.

It will be appreciated that techniques of the present disclosure may beimplemented on any hardware or software platforms suitable for an imagegeneration system. In an exemplary embodiment, the image generationsystem may be implemented on one or more integrated circuits thatconstitute a mobile device, e.g., a smartphone, GPS navigation unit, orother type personal digital assistant. In alternative exemplaryembodiments, the image generation system may be implemented on a desktopcomputer, or on one or more server computers accessible by computingdevices over a wired or wireless network. For example, the generationand storage of immersive views, and generation and superimposition ofdigital signage may all be performed by a server computer, based oninput provided by a user over an Internet connection. Such alternativeexemplary embodiments are contemplated to be within the scope of thepresent disclosure.

Note digital signage as denoted herein may generally display any type ofinformation associated with an entity in an immersive view, e.g.,metadata and/or other features. In certain exemplary embodiments,digital signage may include, e.g., a link to more detailed or specificdirections to an entity than is currently shown (or a link to simpledirections if in a certain instance the digital signage is configurednot to display directions), website, one or more links to an additionalor alternative immersive view associated with an entity, etc. Forexample, digital signage 310 in FIG. 3 or 410 in FIG. 4 may include alink (e.g., a hyperlink associated with the name “First Bank”) whereby auser can, if the link is selected, retrieve onto a display an immersiveview corresponding to an interior view or other view of the building orbusiness. It will be appreciated that one or more retrieved immersiveviews may also incorporate the techniques of the present disclosure,e.g., in generating digital signage in the presence of static or dynamicocclusions. Such alternative exemplary embodiments are contemplated tobe within the scope of the present disclosure.

FIG. 11 illustrates an exemplary embodiment of an apparatus 1100according to the present disclosure. Note FIG. 11 is shown forillustrative purposes only, and is not meant to limit the scope of thepresent disclosure.

In FIG. 11, apparatus 1100 comprises a memory 1110 storing an immersiveview corresponding to viewing parameters comprising a viewing location,and data associated with a preferred entity not displayed in theimmersive view. Memory 1110 further stores a planar layout correspondingto the viewing location. Apparatus 1100 further comprises a digitalsignage generation block 1120 configured to: based on the planar layout,determine a presence of a blocking entity between the viewing locationand the preferred entity; and generate digital signage comprising thedata associated with the preferred entity, the digital signage furthercomprising an indication that the preferred entity is not displayed inthe immersive view if the presence of a blocking entity is determined.Apparatus 1100 further comprises a superimposition block 1130 configuredto superimpose the digital signage onto the immersive view to generatean annotated immersive view.

FIG. 12 illustrates an exemplary embodiment of a computing device 1200according to the present disclosure. Note FIG. 12 is shown forillustrative purposes only, and is not meant to limit the scope of thepresent disclosure.

In FIG. 12, computing device 1200 includes a processor 1210 and a memory1220 holding instructions executable by the processor to: retrieve fromthe memory an immersive view corresponding to viewing parameterscomprising a viewing location, and data associated with a preferredentity not displayed in the immersive view; retrieve from the memory aplanar layout corresponding to the viewing location; based on the planarlayout, determine a presence of a blocking entity between the viewinglocation and the preferred entity; and generate digital signagecomprising the data associated with the preferred entity, the digitalsignage further comprising an indication that the preferred entity isnot displayed in the immersive view if the presence of a blocking entityis determined; and superimpose the digital signage onto the immersiveview to generate an annotated immersive view.

FIG. 13 illustrates an exemplary embodiment 1300 of a method accordingto the present disclosure. Note FIG. 13 is shown for illustrativepurposes only, and is not meant to limit the scope of the presentdisclosure.

In FIG. 13, at block 1310, an immersive view is retrieved correspondingto viewing parameters comprising a viewing location, and data associatedwith a preferred entity not displayed in the immersive view.

At block 1320, a planar layout is retrieved corresponding to the viewinglocation.

At block 1330, based on the planar layout, a presence of a blockingentity between the viewing location and the preferred entity isdetermined

At block 1340, digital signage is generated, the digital signagecomprising the data associated with the preferred entity, the digitalsignage further comprising an indication that the preferred entity isnot displayed in the immersive view if the presence of a blocking entityis determined.

At block 1350, the digital signage is superimposed onto the immersiveview to generate an annotated immersive view.

In this specification and in the claims, it will be understood that whenan element is referred to as being “connected to” or “coupled to”another element, it can be directly connected or coupled to the otherelement or intervening elements may be present. In contrast, when anelement is referred to as being “directly connected to” or “directlycoupled to” another element, there are no intervening elements present.Furthermore, when an element is referred to as being “electricallycoupled” to another element, it denotes that a path of low resistance ispresent between such elements, while when an element is referred to asbeing simply “coupled” to another element, there may or may not be apath of low resistance between such elements.

The functionality described herein can be performed, at least in part,by one or more hardware and/or software logic components. For example,and without limitation, illustrative types of hardware logic componentsthat can be used include Field-programmable Gate Arrays (FPGAs),Program-specific Integrated Circuits (ASICs), Program-specific StandardProducts (ASSPs), System-on-a-chip systems (SOCs), Complex ProgrammableLogic Devices (CPLDs), etc.

While the invention is susceptible to various modifications andalternative constructions, certain illustrated embodiments thereof areshown in the drawings and have been described above in detail. It shouldbe understood, however, that there is no intention to limit theinvention to the specific forms disclosed, but on the contrary, theintention is to cover all modifications, alternative constructions, andequivalents falling within the spirit and scope of the invention.

1. An apparatus comprising: a memory storing an immersive view of anenvironment, a vantage point from which the immersive view is perceived,and data associated with a preferred entity not displayed in theimmersive view from the vantage point; a digital signage generationblock configured to generate digital signage comprising the dataassociated with the preferred entity, the digital signage furthercomprising an indication that the preferred entity is not displayed inthe immersive view; and a superimposition block configured tosuperimpose the digital signage onto the immersive view to generate anannotated immersive view.
 2. The apparatus of claim 1, the memoryfurther storing a planar layout corresponding to the immersive view, thedigital signage generation block configured to: based on the planarlayout, determine a presence of a blocking entity between the vantagepoint and the preferred entity; and generate the indication that thepreferred entity is not displayed in the immersive view if the presenceof a blocking entity is determined.
 3. The apparatus of claim 2, theplanar layout comprising area and vertical height occupied by each of aplurality of entities, the plurality of entities comprising thepreferred entity.
 4. The apparatus of claim 1, the indication that thepreferred entity is not displayed in the immersive view comprisingdirections from the vantage point to the preferred entity.
 5. Theapparatus of claim 1, the superimposition block configured tosuperimpose the digital signage onto the immersive view at a portion ofthe immersive view designated as blank space.
 6. The apparatus of claim1, the superimposition block further configured to superimpose thedigital signage onto a canvas area of the immersive view, the canvasarea calculated to be non-overlapping with at least one dynamicocclusion present in the immersive view.
 7. The apparatus of claim 1,the digital signage further comprising a link to a second immersiveview, the digital signage generation block further configured to receiveuser selection of the link, the apparatus further comprising a displayfor displaying the second immersive view responsive to user selection ofthe link.
 8. The apparatus of claim 1, further comprising a processorconfigured to receive a user query for a query entity type located inthe vicinity of the vantage point, the preferred entity having an entitytype corresponding to the query entity type, the preferred entityfurther being located within a predetermined distance from the vantagepoint.
 9. The apparatus of claim 1, the data associated with thepreferred entity comprising a name of the preferred entity, businesshours, or directions from the vantage point to the preferred entity. 10.The apparatus of claim 1, the data associated with the preferred entitybeing derived from a sponsor of the preferred entity.
 11. A methodcomprising: retrieving an immersive view of an environment, a vantagepoint from which the immersive view is perceived, and data associatedwith a preferred entity not displayed in the immersive view from thevantage point; generating digital signage comprising the data associatedwith the preferred entity, the digital signage further comprising anindication that the preferred entity is not displayed in the immersiveview; and superimposing the digital signage onto the immersive view togenerate an annotated immersive view.
 12. The method of claim 11,further comprising: retrieving a planar layout corresponding to theimmersive view; based on the planar layout, calculating a presence of ablocking entity between the vantage point and the preferred entity; andgenerating the indication that the preferred entity is not displayed inthe immersive view if a blocking entity is calculated to be present. 13.The method of claim 12, the planar layout comprising area and verticalheight occupied by each of a plurality of entities, the plurality ofentities comprising the preferred entity.
 14. The method of claim 11,the indication that the preferred entity is not displayed in theimmersive view comprising directions from the vantage point to thepreferred entity.
 15. The method of claim 11, the superimposingcomprising superimposing the digital signage onto the immersive view ata portion of the immersive view designated as blank space.
 16. Themethod of claim 11, the superimposing comprising superimposing thedigital signage onto a canvas area of the immersive view, the canvasarea calculated to be non-overlapping with at least one dynamicocclusion present in the immersive view.
 17. The method of claim 11, thedigital signage further comprising a link to a second immersive view,the method further comprising: receiving user selection of the link; anddisplaying the second immersive view responsive to user selection of thelink.
 18. The method of claim 11, further comprising: receiving a userquery for a query entity type located in the vicinity of the vantagepoint, the preferred entity having an entity type corresponding to thequery entity type, the preferred entity further being located within apredetermined distance from the vantage point.
 19. An apparatuscomprising: means for retrieving an immersive view of an environment, avantage point from which the immersive view is perceived, and dataassociated with a preferred entity not displayed in the immersive viewfrom the vantage point; means for generating digital signage comprisingthe data associated with the preferred entity, the digital signagefurther comprising an indication that the preferred entity is notdisplayed in the immersive view; and means for superimposing the digitalsignage onto the immersive view to generate an annotated immersive view.20. The apparatus of claim 19, the digital signage further comprising alink to a second immersive view, the apparatus further comprising: meansfor receiving user selection of the link; and means for displaying thesecond immersive view responsive to user selection of the link.